CAPTURING KIDS’ HEARTS 


ABSTRACT 


This report completes a three-year (2016-17, 
2017-18, and 2018-19) quasi-experimental 
evaluation of the fidelity of implementation of the 
Capturing Kids’ Hearts program and its impact on 
two outcomes, teacher-student relationships and 
student behavior within the Wake County 

Public School System (WCPSS). Findings from 
focus groups, teacher self-assessments, and 
classroom observations from Capturing Kids’ 
Hearts schools and comparison schools suggest 
that Capturing Kids’ Hearts is not being 
implemented consistently. Statistical significance 
tests show no difference in the desired outcomes 


between the intervention and comparison schools. 


These findings may be attributed to the fact that 
many WCPSS schools use similar strategies that 
fall under different names. In light of these results, 
the authors discuss implications for practice and 
policy within the district. 
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INTRODUCTION 


During the 2016-17 school year a small number of 
schools within the Wake County Public School 
System (WCPSS) began implementing Capturing 
Kids’ Hearts. This report completes a three-year 
study (2016-17, 2017-18, and 2018-19) which shows 
the fidelity of Capturing Kids’ Hearts 
implementation within 11 WCPSS schools, and 
analyzes and compares the expected outcomes 
between those schools with pre-implementation 
data and their matched comparison schools. Two 
research questions of interest are: 


1) How are teachers implementing Capturing Kids’ 
Hearts, and how does this compare to “business as 
usual” at the comparison schools? 


2) Do the three-year outcome analyses of the 
Capturing Kids’ Hearts intervention show an 
impact on teacher-student relationships and 
student behavior? That is, is there a discernible 
difference between these outcomes at Capturing 
Kids’ Hearts schools and the comparison schools? 


Our study uses descriptive and quasi-experimental 
research designs. As such, the findings will 
facilitate district leaders’ decisions about 
programmatic adjustments to schools currently 
implementing Capturing Kids’ Hearts. Moreover, 
the quasi-experimental research design allows us 
to generalize the study’s findings to schools that 
have similar demographic profiles to the 
Capturing Kids’ Hearts schools so leadership can 
make data-based decisions regarding the feasibility 
of program expansion. 
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Table 1 
Conclusions supported by this study’s research design 


Research Design Conclusions 


We can conclude that the program or policy caused changes in outcomes because the 


. research design used random assignment. 


We can reasonably conclude that the program or policy caused changes in outcomes 


ne a because an appropriate comparison strategy was used. 


Descriptive These designs provide outcome data for the program or policy, but differences cannot 
Quantitative be attributed directly to it due to lack of a comparative control group. 
Qualitative 


Data Source: List, Sadoff, & Wagner (2011) and What Works Clearinghouse (2014) 
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BACKGROUND 


Capturing Kids’ Hearts is a set of processes 
developed by the Flippen Group (2016a) intended 
to create healthy relationships between adults and 
youth and to support high-achieving learning 
environments. It is designed to strengthen students’ 
connection to school by 1) increasing protective 
factors including positive character development, 
strong bonds with teachers, and consistently 
enforced behavioral agreements and 2) decreasing 
risk factors such as inappropriate behavior and poor 
social coping skills (Holtzapple et al., 2011). 
According to the Flippen Group (2016b), Capturing 
Kids’ Hearts aligns with Bandura’s (1977) Social 
Cognitive Theory of Learning which argues that 
behavior is dynamic and dependent on a person’s 
beliefs and environment. Bandura argues that 
people must believe that they can successfully 
master behaviors, increasing self-efficacy before 
they can acquire them. Then, behaviors are acquired 
through observing and imitating the behaviors of 
others and through self-regulation (Perry, Baranowski 
& Parcel, 1991). Following this theoretical 
framework, schoolwide implementation of 
Capturing Kids’ Hearts consists of several 
strategies, collectively referred to as the EXCEL 
Model strategies, used by K-12 classroom teachers: 


* greeting students at the door witha 
handshake 

* asking students to share good things in their 
lives 

* having students create a socia/ contract for 
expected classroom behavior 

* posing four questions to redirect behavior 

* using and encouraging students to use 
non-verbal hand signals to redirect behavior 

* ending the class on a powerful note or /aunch 
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The existing research on Capturing Kids’ Hearts is minimal—it includes a small randomized 
control trial (RCT) conducted by the Flippen Group (Holtzapple et al., 2011). Although RCT 
studies are typically considered the “gold standard” for evidence of effectiveness, this study 
does not fully meet the U.S. Department of Education’s (2016) recommendations for strong 
evidence because it is based on a small sample of eight high schools which restricts its 
generalizability to other school settings. 


WCPSS’ first core belief states that “Every student is uniquely capable and deserves to be 
challenged and engaged in relevant, rigorous, and meaningful learning each day” (Wake 
County Public School System, n.d.). To transform this belief into a reality, it is important that 
WCPSS teachers deliver core instruction and build relationships with students to help reduce 
learning barriers. The Flippen Group believes that strong relationships are critical to learning 
and they market Capturing Kids’ Hearts as an initiative that promotes student engagement 

in positive relationships with each other and their teachers. WCPSS chose to implement this 
initiative as an effort to build positive relationships between teachers and students as a means 
of improving the culture of learning at eight elementary schools, two middle schools, and one 
high school. Figure 1 illustrates the expected implementation and outcome goals of Capturing 
Kids’ Hearts within those schools. 


Figure 1 
Capturing Kids’ Hearts Pathway of Change 


Strategies Implementation Short-Term Long-Term 
Outcomes Outcomes 
« Provide two days Teachers use the Strengthen students’ Improve school 
of training for all EXCEL Model connection to school culture and student 
teachers. strategies: crelationship-building) learning. 
* Train selected * Greeting By 
teachers at each Students * Increasing 
school to be protective factors 
Process * Good Things such as strong 
Champions to : bonds with 
support * Social teachers. 
Contracts 


implementation. 
* Decreasing risk 


. Four Questions 
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* Hand Signals inappropriate 
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To initiate schoolwide implementation, schools and/or districts contract with the Flippen 
Group (2016b) to provide training to teachers and administrators. For an additional cost, each 
school has the option of training a few teachers as Process Champions who serve as mentors 
and facilitate implementation. Contracts with the Flippen Group may also include fees for a 
school visit and online or phone-call implementation support. 


The district’s federally funded Title | program budget has been the primary funding source 
for implementation within WCPSS. Local funds appropriated from the district’s Eastern Area 
budget covered most of the remaining costs. Using these two sources, the district invested 
roughly $750,000 in program implementation over the three-year period. As Table 2 shows, 
the district incurred the majority of expenditures for Capturing Kids’ Hearts in 2016-17, with a 
sharp spending decrease by 2018-19. 


¢ In 2016-17, $615,750 covered training expenses for 14 schools (11 included in our study) and 
for approximately 200 Central Service administrators (see Appendices Table A-1 for a list 
of the 14 funded schools and the 11 included in our study). Typically, costs associated with 
Capturing Kids’ Hearts are highest during a school’s first year of implementation because 
that is when the bulk of program training occurs. 

* Costs were considerably lower in 2017-18, amounting to $106,200 for 13 schools that had 
lower training costs during their second year of implementation. The other school that 
was Originally funded by Title | and local funds applied school redesign funds to support 
implementation (not included in the 2017-18 total). In addition, no Central Service training 
occurred. 

¢ In 2018-19, $24,000 covered the second-year training costs for one school (not included in 
our study because the school opened in 2017-18). 


Table 2 
WCPSS spent nearly $750,000 on Capturing Kids’ Hearts over 3 years 


Federal Title | Local Total 

2016-17 $493,200 $122 550 $615,750 

Training, school visits/support and 2017-18 $78,200 $28,000 $106,200 
travel expenses for trainers i 5 i ish A 
2018-19 $24,000 $24,000 

Totals $745,950 


Data Source: WCPSS Title | Department and the Eastern Area. 


Note: The cost for the initial training session for 50 participants is about $21,000, plus 
$400-450 for each additional participant (not to exceed 60), in addition to travel 
expenses for the facilitator billed at a rate of $1,000 for one-day events, $1,500 
for two-day events, and $1,800 for three-day events. School redesign funds are 
not included in totals. 
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METHODS 


This study utilized descriptive data to measure 
implementation of the Capturing Kids’ Hearts 
intervention during the 2016-17, 2017-18, and 
2018-19 school years and applied a 
quasi-experimental research design to compare 
outcomes between intervention and comparison 
schools across the same years. We - the authors of 


this study - used a statistical technique! to match 
comparison schools to intervention schools, first 
comparing the individual student-level variables 
within all possible school-level matches, and then, 
from that pool, selecting schools that were most 
similar to the Capturing Kids’ Hearts schools. 
District leaders chose to implement the program 
during one elementary school’s first year of 
operation, so 10 of the 11 Capturing Kids’ Hearts 
schools in this study had pre-implementation 
data (2015-16) on which to match 10 comparison 
schools. Our examination of the schools before 
and after the matching procedure found that they 
were well balanced in terms of school-level pretest 
outcomes, percentages of students from 
economically disadvantaged households, 
suspensions, and membership counts. 
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We used the following data sources to answer our research questions: 


* In 2016-17, we asked staff from the Data, Research & Accountability (DRA) Department to 
help us facilitate a total of 17 focus groups at 11 Capturing Kids’ Hearts schools with 
principals and small random samples of teachers and students to assess the fidelity of 
implementation. 

* |n 2017-18 and 2018-19, we asked teachers at the Capturing Kids’ Hearts schools to 
complete an implementation self-assessment each spring. Almost 300 teachers 
participated in the spring of 2018 and about 450 teachers responded in the spring of 2019. 

¢ In 2017-18, DRA staff also helped us conduct short classroom walk-throughs at the 11 
Capturing Kids’ Hearts schools and 10 comparison schools to observe teacher and student 
engagement. Overall, DRA staff visited approximately 100 classrooms at the intervention 
schools and 100 at the comparison schools. 

¢ Finally, we used teacher-student relationship and student behavior data collected from 
2015-16 through 2018-19 to conduct multi-level analysis of impact on the 10 intervention 
schools with pre-implementation data and their matched counterparts. 
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RESULTS 


HOW ARE TEACHERS 
IMPLEMENTING CAPTURING KIDS’ 
HEARTS, AND HOW DOES THIS 
COMPARE TO “BUSINESS AS 
USUAL” AT COMPARISON 
SCHOOLS? 


Within the 2017-18 WCPSS Teacher Survey, 572 
teachers and 122 instructional coaches at all 
Capturing Kids’ Hearts schools responded to 
questions about the Capturing Kids’ Hearts 
training. Of those respondents, 91% told us they 
received training led by the Flippen Group with 
an average of five teachers and/or instructional 
coaches per school attending the training for 
Process Champions. During the focus group 
discussions, many principals and teachers shared 
positive feedback about the training sessions and 
said that they want training for all school staff (not 
just teachers and administrators), timely training 
for newly-hired teachers, and refresher training 
sessions throughout the year. 


Nevertheless, convergent findings from focus 
group interviews, teacher self-assessments, and 
classroom observations suggest that overall 
Capturing Kids’ Hearts schools have not 
consistently and fully implemented the EXCEL 
Model strategies, especially at the middle and high 
schools. Upon checking for variation in 
implementation across schools, no single school 
stood out as an example of full implementation of 
all program components. In addition, the 
employed EXCEL Model strategies do not appear 
distinctly different from the “business-as-usual” 
strategies that comparison schools used in their 
efforts to build relationships and positive school 
cultures. 
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Greeting Students 


Within 10 of the 11 focus group interviews 
conducted in 2016-17, teachers at Capturing 
Kids’ Hearts schools stated that they like 
greeting students at the door and believe 
this to be an effective strategy for 
connecting with students. Teachers 
mentioned that greeting students is a great 
way to start the day and helps them tune 
into how the students might be feeling. 
Students at Capturing Kids’ Hearts schools 
often said that they like this practice and that 
it makes them feel welcomed, however, some 
students perceived the practice to be 
inauthentic and others said that their 
teachers don’t greet them at all. Based on 
Capturing Kids’ Hearts self-assessment 
survey results collected in the spring of 2018 
and 2019, almost all teachers reported 
greeting students with a pleasant tone and 
welcoming body language (>97%). Teachers 
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also reported that they typically greet 
students by name (about 95% for elementary 
and approximately 90% for middle and high 
school teachers, each year). 


During classroom observations conducted in 
2017-18, DRA staff noticed that school 
personnel at Capturing Kids’ Hearts schools 
as well as comparison schools were 
generally welcoming to students and to staff, 
with a few exceptions. DRA staff observed 
some type of morning greeting at each of 
the eight Capturing Kids’ Hearts elementary 
schools and each of the seven comparison 
elementary schools, though there was more 
consistency at some schools than others. 
Similar greetings, teachers saying, “Good 
Morning” and speaking with students about 
how they were doing, occurred at 
elementary Capturing Kids’ Hearts schools 
and the elementary comparison schools. 


all 


Greetings were also common at the two 
Capturing Kids’ Hearts middle schools and 
the high school’s Freshman Academy but not 
at the middle and high school comparison 
sites. Contrary to teachers’ self-assessment 
results, the observations at the Capturing 
Kids’ Hearts schools did not consistently 
show teachers greeting students by name, 
although this practice occurred more than 
what was observed at the comparison 
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schools. It was rare to see teachers at both 
the Capturing Kids’ Hearts and comparison 
schools greet students with a hand shake 
or other gesture, except at one elementary 
comparison school that was implementing 
the Leadership Model Program based on 
Covey’s 7 Habits of Highly Effective People? 
where teachers and students consistently 
shook hands. 


ae 


Good Things or Connection Activities 


Teachers at Capturing Kids’ Hearts schools 
expressed positive feelings about giving 
students the opportunity to share their Good 
Things in class within nine of the 11 focus 
groups. Overall, they felt that the practice 
helps them get to know their students. Their 
students’ feelings were more variable. For 
instance, buy-in of this practice appears to 
be better at the elementary level, where 
students said it makes them feel good and 
special. At the middle and high school levels, 
however, students described it as 

something silly or uncomfortable. The 
majority of elementary, middle, and high 
school teachers at Capturing Kids’ Hearts 
schools (>75% in spring of 2018 and 2019) 
reported that they typically invite students to 
share Good Things and ask them follow-up 
questions. 
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Within the Capturing Kids’ Hearts schools, 
DRA staff observed the Good Things 
activity in at least one classroom at seven of 
the eight elementary schools and one 
middle/high school. During the vast 

majority of these observations, teachers 
engaged with students during this activity 
by asking follow-up questions or practicing 
active listening. Comparison school teachers 
used similar activities to connect and engage 
with students at five of the seven 
elementary schools and at one school at the 
middle/high school level. These activities 
were more diverse across teachers than the 
more standardized implementation of Good 
Things at the Capturing Kids’ Hearts schools, 
yet they appeared to serve the same 
purpose. For instance, teachers might have 
given students time to share personal stories 
that were related to the topic of instruction. 


Social Contract or Behavioral 
Expectations 


At six of the 11 focus groups, mostly at the 
elementary level, teachers in Capturing Kids’ 
Hearts schools identified the Social Contract 
as effective in building student 
accountability and ownership. They also 
discussed the importance of revisiting the 
Social Contract and modifying it throughout 
the year. Students talked about the Social 
Contracts in their classrooms which might 
include declarations such as “be kind,” 
“follow directions,” or “listen when others are 
speaking” as well as other statements related 
to respect and kindness. Nearly all teachers 
at the 11 Capturing Kids’ Hearts schools 
reported that they developed a Social 
Contract with their students and posted it in 
a prominent place in the classroom (>95% 

in spring of 2018 and 2019). These teachers 
were less likely to report that they use the 
contract, however. Each year, about 80% of 
the elementary school teachers said they 
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refer to the Social Contract when a student is 
violating it, and slightly smaller 

percentages responded that they ask for 
feedback concerning how well the class is 
abiding by it (73% in spring of 2018 and 64% 
in spring of 2019). At the middle and high 
school level, considerably lower percentages 
of teachers reported using these strategies 
each year (about half refer to the contract 
when it is violated and about one third ask 
for feedback about it). 


cnt, 
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Classroom observations data corroborated 
the presence of Social Contracts as DRA 
staff saw them posted consistently in 
classrooms throughout all but one of the 
Capturing Kids’ Heart schools. Classrooms 
within seven of these schools (five of the 
elementary and two of the middle/high 
schools) displayed Positive Behavioral 
Interventions and Supports (PBIS) behavioral 
expectations in addition to the Social 
Contracts. PBIS is a framework for using 
interventions and its systems and structures 
of behavioral expectations and 
consequences promote positive school 
climates and strong teacher-student 
relationships. DRA staff did observe a couple 
of teachers reviewing the Social Contract to 
either frontload expectations or redirect 
student misbehavior. Five of the eight 
Capturing Kids’ Hearts elementary schools 
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had discipline plans or consequences posted 
in most of their classrooms which related to 
either Capturing Kids’ Hearts or PBIS. At one 
elementary school, it appeared to be part 

of the school culture for students to greet 
classroom visitors and ask them to sign the 
Social Contract. When asked to explain the 
Social Contracts, students almost always 
described them as the class rules that 
students were expected to follow or else 
there would be a consequence. DRA staff did 
not see consequences posted at the 
Capturing Kids’ Hearts middle and high 
schools. 


At the comparison schools, DRA staff 

noted PBIS behavioral expectations 
displayed throughout six of the 

elementary schools and all three middle and 
high schools. There were also a few examples 
of social contracts and additional classroom 
rules. None of the comparison schools had 
posters displaying consequences for 
misbehavior. 


Four Questions, Hand Signals, and Other 
Student Behavior Management 


Capturing Kids’ Hearts teachers in 10 of the 
Tl focus groups discussed the Four 
Questions as the most challenging EXCEL 
Model strategy. They said that it takes time 
away from instruction and that it does not 
always work, especially for students who 
repeatedly exhibit problem behaviors. The 
vast majority of students who 

participated in the focus groups could 
articulate the consequences at their schools, 
which usually included the Four Questions. 
Implementation of this strategy appeared to 
be inconsistent as the students frequently 
mentioned that some teachers used it and 
others did not. In the spring of 2018 and 
2019, about two thirds of Capturing Kids’ 
Hearts elementary school teachers reported 
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that they typically ask the Four Questions. 
Among middle and high school teachers in 
2018, 41% stated that they generally ask the 
Four Questions, and this percentage rose to 
47% in 2019, although the increase was not 
statistically significant. 


Another challenging strategy mentioned 

by teachers at nine of the 11 Capturing Kids’ 
Hearts schools is the hand signals, called 
“checks” and “fouls.” Teachers discussed 

the ineffectiveness of this practice in lower 
grades where it can get overused as well as 
in the upper grades where students think 

it is “uncool.” Across the grade spans, the 
teachers and students gave examples of 
how the use of the hand signals can be too 
aggressive. Within each of the Capturing 
Kids’ Hearts self-assessment surveys we 
asked teachers about their use of non-verbal 
signals for classroom management, which 
could be interpreted as “checks” and “fouls” 
and/or the teacher’s hand-signal to refocus 
students’ attention. In the spring of 2018 and 
2019, most elementary school teachers (90% 
and 85%) reported using non-verbal signals 
whereas each year about half of middle and 
high school teachers did so (they were 
moving in a positive but not statistically 
significant direction across the years). 


When observed during a typical school day, 
Capturing Kids’ Hearts and comparison 
school students were generally on-task and 
seemed comfortable interacting with each 
other and their teachers. DRA staff noticed 


very few student behavioral issues. When the 


situation warranted it, teachers at both 
Capturing Kids’ Hearts and comparison 
schools used a variety of strategies, 
including hand signals, to gain students’ 
attention or manage behavior. DRA staff 
saw a few teachers at Capturing Kids’ Hearts 
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schools asking the Four Questions, although 
there were also instances when teachers 

did not use this strategy to redirect student 
behavior. 


Launch 


At Capturing Kids’ Hearts schools, there was 
very little discussion about the Launch within 
the focus groups and very few opportunities 
to observe its application during classroom 
visits. Elementary teachers’ practice of 
ending the class on a powerful note appears 
to have moved in a positive, albeit not 
statistically significant direction between the 
spring of 2018 and 2019 (48% and 57%, 
respectively). A much higher percentage of 
the middle and high school teachers 
reported using this strategy (about 70% 
each year). Similarly, DRA staff did note that 
teachers closed with a Launch in each of 

the classrooms observed at the high school 
implementing Capturing Kids’ Hearts. 


Visual summaries of the findings from the 
diverse data sources are presented below 
(see Tables 3 and 4 and Figures 2 and 3). 
More extensive details can be seen in Tables 
A-2 to A-4 (Appendix A). 
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Table 3 


In 2016-17, teachers reported more positive perceptions of Capturing Kids’ Hearts 
implementation than students 


EXCEL Model Strategies Teacher Perceptions Student Perceptions 
Greeting Students Effective Mixed views on effectiveness 
Good Things Effective Mixed views on effectiveness 
Social Contract Effective Less effective 
Four Questions Less effective Less effective 
Hand Signals Less effective Less effective 
Launch Very little discussion of this strategy by either group 


Data Source: Teacher Focus Groups at 11 Capturing Kids’ Hearts Schools and Student Focus Groups at 
four Capturing Kids’ Hearts Schools (2 elementary, 1 middle and 1 high), 2016-17 


Table 4 


In 2017-18, teachers within both groups of schools used similar strategies, yet use was 
inconsistent 


EXCEL Model or Similar Strategies Capturing Kids’ Hearts Schools Comparison Schools 
approx. 100 classrooms approx. 100 classrooms 

Greeting Students Moderately consistent Moderately consistent 
Good Things or Connection Activities Moderately consistent Moderately consistent 
Social Contract and/or PBIS 

+ Behavioral Expectations Displayed Consistent Consistent 

« Behavioral Consequences Displayed Inconsistent None 
Four Questions, Hand Signals, or Other ; . 
Student Behavior Manag Moderately consistent Moderately consistent 

Inconsistent Very few opportunities to view 

Launch or dismissal * very few opportunities to view 


* seen only at the high school 


Data Source: Classroom Observations at Capturing Kids’ Hearts Schools (8 elementary, 2 middle and 1 
high) and Comparison Schools (7 elementary, 2 middle and 1 high), 2017-18 
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Figure 2 


Elementary teachers’ self-assessment results indicated no improvement in implementation of 
Capturing Kids’ Hearts between the spring of 2018 and 2019 


Difference 


Welcoming body language when greeting @ 
Pleasant tone when greeting co) 
Greet students by name @ 
Social contract prominently posted eo 
Developed social contract eo 
Ask follow up questions after asking about good things ®@ 


Use non-verbal signals for self-managing class o-_ 


Refer to social contract when not behaving 
Ask about good things eo 
Ask how well is class abiding by social contract eo 


Ask four questions for misbehavior eo 


Dismiss students with motivational message —#o 
25% 50% 75% 100% 


Percentage of Teachers 


Note: Difference indicates the percentage point difference in teachers who reported that they always/ 
almost always or often implement the various EXCEL Model strategies based on the results 
of our survey conducted in spring of 2018 and spring of 2019. Green highlights show positive 
changes across the years. None of the differences were statistically significant. 


Capturing Kids’ Hearts 17 


Wake County Public School System 


Figure 3 


Middle and high school teachers’ self-assessment results indicated no improvement in imple- 
mentation of Capturing Kids’ Hearts between the spring of 2018 and 2019 


Pleasant tone when greeting = = 6 aa 
Welcoming body language when greeting C J 

Social contract prominently posted eo. 
Developed social contract oe. 

Greet students by name e@ 

Ask follow up questions after asking about good things eo. 

Ask about good things eo 
Dismiss students with motivational message eo 
Use non-verbal signals for self-managing class i 
Refer to social contract when not behaving e_ 
Ask four questions for misbehavior e-e 
Ask how well is class abiding by social contract oO 
25% 50% 75% 100% 
Percentage of Teachers 


Note: Difference indicates the percentage point difference in teachers who reported that they always/ 
almost always or often implement the various EXCEL Model strategies based on the result of 
our survey conducted in spring of 2018 and spring of 2019. Green highlights show positive 
changes across the years. None of the differences were statistically significant. 
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DOES CAPTURING KIDS’ HEARTS HAVE AN IMPACT ON TEACHER 
AND STUDENT RELATIONSHIPS AND STUDENT BEHAVIOR? 


Findings from student and teacher surveys and records of student suspensions suggest that 
schools implementing Capturing Kids’ Hearts have reported levels of student relationships and 
behavior that are no different than comparison schools implementing other programs. Over 
time, these two groups of schools have shown similar trends in these outcomes. 


Relationships and Behavior Outcomes 


Capturing Kids’ Hearts intends to strengthen students’ connection to school by increasing 
students’ protective factors such as strong bonds with teachers and decreasing their risk 
factors such as inappropriate behavior and poor coping skills. These outcomes should in turn 
improve the school culture. Measures for this study included districtwide survey data based 
on 1) student reports of their relationships with teachers and peer support for learning? and 
teacher reports of student self-management skills* as measures of protective factors and 
behavior and 2) school suspension rates as measures of risk factors. Analyses did not reveal 
any programmatic impact on the desired outcomes, as illustrated in Figures 4-6. Additional 
analyses were conducted to examine whether there were any program effects for various 
subgroups, but none were found. 


Teacher and student survey results reported before (2015-16) and after implementation show 
little change among the various outcomes. Mean ratings of student perceptions of 
relationships with their teachers and peer support for learning were not different between 
Capturing Kids’ Hearts schools and comparison schools. No detectable differences were found 
for teacher reports of student relationships and self-management skills at Capturing Kids’ 
Hearts and comparison schools. Likewise, both Capturing Kids’ Hearts schools and the 
comparison schools have shown a downward trend in short-term and long-term suspensions, 
with no statistically significant difference between the groups’ suspension rates. The district 
implemented other programs and initiatives to reduce suspensions during the same time 
frame of this study, which may have contributed to the overall decline seen at both sets of 
schools. 


Capturing Kids’ Hearts 19 


Figure 4 


Wake County Public School System 


According to students, relationships have not improved over time 


Teacher-Student Relationship 
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Peer Support for Learning 
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2017-18 


Data Source: Student Engagement Instrument Results, 2015-16 to 2018-19 


Note: 


Figure 5 
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2015-16 represents baseline data before 2016-17 implementation of Capturing Kids’ Hearts. No 
statistical difference between groups or by race/ethnic subgroups. 


According to teachers, student behavior has not improved over time 


Students Communicate in Respectful and 
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Students Take Responsibility for 
Their Actions 


School Type 

-®- Comparison 

-® CKH 
2017-18 2018-19 


2015-16 2016-17 


Data Source: North Carolina Teacher Working Conditions Survey and WCPSS Teacher Survey Results, 


2015-16 to 2018-19 


Note: 


2015-16 represents baseline data before 2016-17 implementation of Capturing Kids’ Hearts. Two 
Capturing Kids’ Hearts schools and one comparison school had teacher response rates below 


33% on the teacher survey in 2018-19; thus, responses do not reflect the schools’ teachers’ 
perceptions overall. No statistical difference between groups. 
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Figure 6 


Suspensions have declined at both Capturing Kids’ Hearts and comparison schools over time 
at a similar rate 
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2015-16 2016-17 2017-18 2018-19 2015-16 2016-17 2017-18 2018-19 
Data Source: WCPSS Student Incident Report System, 2015-16 to 2018-19 
Note: 2015-16 represents baseline data before 2016-17 implementation of Capturing Kids’ Hearts. No 


elementary students received long-term suspensions. No statistical difference between groups 
or by race/ethnic subgroups. 
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CONCLUSIONS AND 
RECOMMENDATIONS 


The findings of this three-year study suggest that 
Capturing Kids’ Hearts is not being implemented 
to complete fidelity. Convergent results from focus 
group interviews, teacher self-assessments, and 
classroom observations imply that some EXCEL 
Model strategies have become a greater part of 
the school culture than others. For instance, 
teachers appear to be more effectively and 
consistently greeting students, listening to 
students share their Good Things, and posting 
Social Contracts in the classroom than they are 
managing behavior through the use of Social 
Contracts, the Four Questions, and Hand Signals. 
This could be because Greetings, Good Things, 
and posting Social Contracts are strategies that 
are more straight-forward and easier to imple- 
ment than the others. Elementary schools appear 
to have more fully implemented the EXCEL Model 
strategies than middle and high schools in part 
because teachers perceive these strategies to be 
more appropriate for elementary students, 
particularly the lower grade levels. This perception 
was a common theme heard during focus group 
discussions with Capturing Kids’ Hearts teachers. 


To date, the goals of Capturing Kids’ Hearts have 
not been fully met. As shown in Figure 7, the 
implementation of all EXCEL Model! strategies 
appears to be an area in need of improvement. We 
checked for variation in implementation across 
the schools and found that implementation results 
were largely similar across schools. Elementary 
schools’ levels of implementation were very 
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similar to each other as were the middle and high schools, with little significant improvement 
over the course of our study. Our investigation of teacher training in 2017-18 indicated that 
each school had Process Champions in place who could spearhead school implementation, 
revitalize current staff, and train new staff; therefore, teacher turnover should not have been a 
primary reason for lower than expected implementation levels. 


Figure 7 


Overall, the goals of Capturing Kids’ Hearts have not been fully met 


2016-17 2017-18 2018-19 
Training 
Train all teachers N/A ae N/A 
Train selected teachers at each school to be N/A | fy N/A 


Process Champions 
2016-17 to 2018-19 


Implementation of Exce/ Model Strategies 


Greeting Students Hl 

Good Things Hl 

Social Contracts ll 

Four Questions || 

Hand Signals | | 

Launch ea 
Outcomes 


Student relationships and behavior 


Not Met Fully Met 
eeeee B..... . B..... ame 


Note: N/A indicates that data were not collected that year. Goal status ratings ranging from “Fully 
Met” to “Not Met” visually represent the general extent to which implementation and outcomes 
have been achieved as evident in the findings synthesized from all data sources and analyses 
applied within the study. 


Analyses were not able to detect a statistically significant effect of the Capturing Kids’ Hearts 
program, which could suggest that either the program had weak or negligible impact on the 
measured outcomes or simply that many WCPSS schools use similar strategies that fall under 
different names and which produce similar results. Social and emotional learning (SEL) is an 
umbrella for prosocial skill development and various types of educational interventions such 
as character education, conflict resolution, and bullying prevention (Humphrey et al., 2011; 
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Social and Character Development Research 
Consortium, 2010). The strategies, goals, and 
outcomes of these SEL interventions may be 
comparable to the Capturing Kids’ Hearts 
EXCEL Model framework. Although the 
EXCEL Model strategies are distinctly named, 
the intentions underlying these strategies 
may not be inherently unique to the program. 
They tend to resemble other SEL 
interventions and what is considered to 

be “business as usual” at the comparison 
schools. For example, elementary schools are 
generally places in which teachers greet 
students, encourage students to share 
personal stories and make connections, and 
create expectations and positively manage 
student behavior. Overall, our Capturing Kids’ 
Hearts and comparison elementary schools fit 
this description. Teachers at Capturing Kids’ 
Hearts middle and high schools, however, 
more consistently greeted students than their 
counterparts; otherwise the school 
environments were very similar. 


The two groups of schools may also look 
similar because Capturing Kids’ Hearts has 
not been implemented to a level that would 
generate a difference and/or because other 
SEL programs are also being implemented in 
both Capturing Kids’ Hearts and comparison 
schools. According to principals, in addition 
to implementing Capturing Kids’ Hearts in 
2017-18, all but one school were 
implementing or beginning to implement 
PBIS, three schools were implementing 
Character Education, and at two schools, 
counselors provided SEL training to teachers 
and support to students. Nine of the 10 
comparison schools were implementing PBIS, 
two schools were participating ina 
relationship-building program called The 
Positivity Project®, one school was using the 
Leadership Model Program, and one school 
was providing other SEL support to students. 
Given this list of SEL programs, it is not 
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surprising that this study did not find a 
distinct impact of Capturing Kids’ Hearts on 
relationships, behavior, and the school 
culture in general. As such, we did not 
further investigate how these outcomes 
might impact student learning because any 
differences could not be attributed to 
program implementation. 


Our analysis was based on a sample size that 
may not have been large enough to detect 
slight differences between the two groups. 
The small number of schools (10 
implementing Capturing Kids’ Hearts and 

10 comparison schools) did not generate 
effect sizes that were sufficient enough to 
show important differences between those 
implementing the program compared to 
those implementing other best practices and 
strategies. Additionally, our data measuring 
outcomes do have limitations. The district 
administers the Student Engagement 
Instrument (Betts et al., 2010) each spring to 
measure several domains that serve as 
outcome measures to various program 
interventions; however, prior to 2018-19 
(when the district moved to surveying 
students in grades 3-12) schools were only 
required to survey students in grades 5, 8, 
and 9. In 2017-18, several WCPSS schools did 
not participate in the survey (including one 
of our study schools). As such, data prior 

to 2018-19 reflect one grade level within the 
school not groups of students participating 
in specific programs nor the entire school 
population. 


Ultimately, this study’s findings of limited 
implementation and no discernable 
differences in outcomes between Capturing 
Kids’ Hearts schools and comparison schools 
can likely be attributed to many WCPSS 
schools using similar strategies resulting 

in similar outcomes. The district is seeking 
to adopt programs and strategies that are 
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backed by strong evidence and preferably, do not overlap with similar programs already being 
implemented. The findings presented in this report coupled with the relatively high cost of 
Capturing Kids’ Hearts compared to other SEL-related initiatives offer no support for starting 
implementation at additional schools within the district, yet there appears to be no harm in 
continuing implementation at current schools. Accordingly, we offer the following 
recommendations: 


Recommendation 1: Continue implementation at current schools and dialogue with Capturing 
Kids’ Hearts principals and teachers to identify potential barriers to consistent and authentic 
use of all EXCEL Model! strategies and offer support to improve the fidelity of schoolwide 
implementation for schools. 


Recommendation 2: In lieu of Capturing Kids’ Hearts expansion, investigate less costly 
initiatives that have the potential to or already show evidence of promoting teacher and 
student relationships, keeping in mind those which are already being implemented in WCPSS. 
Also, consider that some initiatives may be more appropriately geared toward elementary 
students than middle and high school students and vice versa. 


Recommendation 3: Continue support for SEL data collection instruments to use throughout 
the district to better support strategic decision-making around those types of initiatives. 
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ENDNOTES 


' The cluster-randomized trial (CRT) provides 
unbiased estimates of the intervention’s impact. 
In a CRT, randomization occurs at the cluster, in 
this case school-level, allowing all students in the 
school to participate in the schoolwide program, 
which makes it possible to be confident that the 
intervention and control schools are equivalent in 
expectation of receiving the treatment. This study 
uses the CRT design procedure for 

matching comparison schools to intervention 
schools. To see details, contact us for the 
Technical Report at: 


wcpssdataaccountab@wepss.net 


2 The Leadership Model Program develops 
leadership in young children. Using Stephen 
Covey’s 7 Habits of Highly Effective People, 
students learn and practice integrity, initiative, 
work ethic, cooperation, confidence, and 
community service. 


3 Within the Student Engagement Instrument 
(Betts, Appleton, Reschly, Christenson, & Huebner, 
2010), students answer questions about 
relationships and peer support on a scale ranging 
from 1 to 4 (3 and 4 indicate positive views). 


4 Within the Teacher Surveys, teachers respond 
to statements about students based on the same 
four-point scale. 


° The Positivity Project builds positive 
relationships by giving teachers the training, 
strategies, and resources to teach positive 
psychology’s 24 character strengths. 
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Table A-1 
Fifteen schools implemented Capturing Kids’ Hearts and 11 were included in the study 


Wake County Public School System 


12. 


13. 


14. 


15. 


School 


East Wake High 


Wendell Middle 


Zebulon Middle 


Beaverdam Elementary 


Knightdale Elementary 


Wakelon Elementary 


Zebulon Elementary 


Brentwood Elementary 


East Garner Elementary 


. Lynn Road Elementary 


. Wilburn Elementary 


Knightdale High 


East Wake Middle 


Creech Road Elementary 


Rogers Lane Elementary 


1st Year of 
Implementation 


2016-17 


2016-17 


2016-17 


2016-17 


2016-17 


2016-17 


2016-17 


2016-17 


2016-17 


2016-17 


2016-17 


NANA ANA AA 


N A 


Included in Three Year Study 


Not included in study, implementation started in 2015-16. Used 
2015-16 School Improvement Funds for implementation so not included 


in Table 2 of expenditures. 


2015-16 Not included in study, implementation started in 2015-16 


2016-17 Not included in study, implementation started late in 2016-17 


2017-18 Not included in study, implementation started in 2017-18 


Capturing Kids’ Hearts 


28 


Wake County Public School System 


Table A-2 


In 2016-17, teachers perceived the implementation of some Capturing Kids’ Hearts 
strategies to be effective and some to be challenging 


EXCEL Model! Strategies Supporting Statements and Quotes from Teachers 


Most Effective Greeting Students | like shaking their hand at the door every day because it starts us off with smiles. It 
sets the tone more positively in the classroom. It builds mutual respect. 
It is good because you notice every child as they come in when before you may 
have not noticed if a child was having a bad day. 
| like being at the door and greeting them because | get a sense of how they are that 
day and can feel them out for the day. 

Good Things 

Yes, they all want to talk, even those who usually don't talk. 


| enjoy the sharing of good things because we get to know things about the students 
that we probably wouldn't know otherwise. 


It gives us the opportunity to let them know we care. We get time to hear and share 
something about ourselves so they learn about us too. 


Social Contract — The Social Contract is great with kindergarten students. | just have to point to it and 
they get it. 
The Social Contract is most effective, however, it is necessary to revisit it 
throughout the year and perhaps rewrite it midyear. 


The Social Contract is a great visual. The kids own it and know they need to follow 


it because they said they would. 
Least Effective The Four ; : ; : 
or Most Questions The questions take too much time and give attention to one student. 
Challenging When you pull a child aside to ask the Four Questions, you lose the rest of the 
class. 


If | put attention on one student then the next 20 will be doing the same and I'll be 
going around the room asking the Four Questions. 


We get to that fourth question and they know there's not a consequence that means 
anything...they’re laughing at us. 


Hand Signals The fouls get overused which make students feel not part of a community. The 
(Checks and younger kids use it, but the older kids would never. 
Fouls) 


The hand signals are elementary concepts. 


In fifth grade it is uncool to check other students, so no one uses. 


Middle school students have a difficult time buying into the hand signals. ..they don't 
think they are cool. 


The checking can turn into arguments. 


Kids want to use the ‘thumb signal’ [check] in people's faces. 


Data Source: Teacher Focus Groups at 11 Capturing Kids’ Hearts Schools, 2016-17 
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Table A-3 


In 2016-17, students perceived the implementation of some Capturing Kids’ Hearts 
strategies to be effective and some to be challenging 


EXCEL Model Strategies Supporting Statements and Quotes from Students 


Mixed Greeting It makes me feel good. It makes me feel like I’m welcome to be in the classroom. 


Effectiveness = Students | feel like I'm very special because they're shaking my hand | feel like this is better 
to like greet someone. 


It makes me feel like | am going to have a good day. 
It is silly. 
It seems fake. 


Teachers don't do anything to make us feel welcome upon entering the classroom; 
they simply take attendance and then get on with their lessons. 

Good Things You get to share with other people and other people get to share their good things 
and their good thing might be close to your good thing. 
When | say my good thing | just feel really good because | get to share the good 
things going on in my life. 
Sometimes | feel uncomfortable sharing because | feel it is not someone's 
business. 
It's stupid. There’s nothing good to share. 
We're only allowed to share good things, but my life is all bad. That's okay. We 


should be allowed to share the bad things that happen to us, because those are 
the things that are funny, and what would make us bond as a class. 


Least Social Kids act the same way regardless of whether there's a piece of paper on the wall 
Effective or Contract telling them how they're supposed to act. 


Most We used them a lot at the beginning of the year, but we barely talk about the social 
Challenging contract now. 


Teachers don't follow it. 
The kids don't pay attention to it. 
The Four My teachers don't ask them, but they are on the board. 


Cnet Our teachers, they don't do the Four Question thing, they go right into the 
punishment. 


My class uses it. If someone gets in trouble, the teacher asks ‘What are you 
doing? What are you supposed to be doing?’ 


It is also on a poster on the wall. 


Hand ‘Foul’ means someone is being mean and ‘check’ means ‘shush.’ Check isn't 
Signals used at all. If you get fouled, you have to say two nice things about the person who 
(Checks and fouled you. It makes you feel like you're in time-out, like you're still in kindergarten. 
Fouls) Most of the time some people use them for weapons. 


Most of the time | don't like them. We have ‘tiny teacher’ who calls on people to do 
Good Things, but sometimes they check people and hit people in the head. 


Data Source: Student Focus Groups at Capturing Kids’ Hearts Schools (2 elementary, 1 middle and 1 high 
school), 2016-17 
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In 2017-18, teachers within both groups of schools used similar strategies, yet use was 


inconsistent 


Capturing Kids’ Hearts Schools (N=11) 
8 elementary, 2 middle, 1 high school 
approx. 100 classrooms 


Comparison Schools (N=10) 
7 elementary, 2 middle, 1 high school 


approx 100 classrooms 


Greeting 
Students 


Moderately Consistent: 


* Some type of morning greeting occurred at each of 
the 8 elementary schools. At 4 schools, teacher 
greetings occurred across all grade levels. 
Greetings varied by grade at the other 4. 


* Greetings prevalent at the middle schools and the 
Freshman Academy at the high school. 


+ At 3 of the elementary and 2 of the middle/high 
schools, the teachers regularly greeted students by 
name. 

* Afew elementary teachers shook hands with 
students or gave high fives or hugs. Several 
middle/high teachers used hand gestures, such as 
handshakes, fist-bumps, elbow-bumps, and high 


Moderately Consistent: 


* Some type of morning greeting occurred at each of 
the 7 schools. At 5 schools, teacher greetings 
occurred across all grade levels. Greetings varied 
by grade at the other 2. 


« Greetings not common at middle/high school. 


« A few instances of teachers greeting students by 
name at the elementary level and none observed at 
the middle/high school. 

* Several elementary and a couple of middle school 
teachers used hand gestures, such as 
handshakes, fist-bumps, elbow-bumps, and high 
fives or gave hugs. 


fives or gave hugs. 


Moderately Consistent: 


« Teachers used some type of activity to connect and 
engage with students at 5 of the elementary 
schools and at 1 middle/high school. The types of 
activities were more diverse compared to the 
implementation of Good Things as a Capturing 
Kids’ Heart common practice. 

Consistent: 


+ PBIS behavioral expectations posted throughout 6 
of the elementary schools and all 3 middle/high 
schools. A few examples of Social Contracts and 


None: 


* None of the comparison schools had posters 


displaying consequences for misbehavior. 


schools but not seen at middle/high schools. 


Good Things Moderately Consistent: 

Or Connection * Teachers used Good Things in at least one 

Activities classroom at 7 of the elementary schools and at 1 
middle/high school. During the vast majority of 
observations, teachers engaged with students 
during this activity by asking students follow-up 
questions or practicing active listening. 

Behavioral Consistent: 

Expectations * Social contracts posted in classrooms throughout 7 

Displayed of the elementary schools and all 3 middle/high 
schools. Some teachers also posted additional 

* PBIS behavioral expectations also displayed 
throughout 5 elementary and 2 middle/high schools. 
+ Afew teachers made reference to the Social 

Contract and at 1 elementary school student 
greeters asked DRA observers to sign it. 

Consequences Inconsistent: 

Displayed * Consequences posted throughout 5 elementary 

Four Moderately Consistent: 

Questions, « “Hand Signal” posters displayed and teachers used 

Hand Signals, a time-out hand signal and other strategies to gain 

and Student students’ attention at 5 elementary schools and 2 

Behavior middle/high schools. 

Management 


* Few observations of students misbehaving. Some 
elementary teachers referred to the Four Questions 
or Social Contract to redirect behavior and others 
did not. No observations of students using 


Moderately Consistent: 


* Teachers used various strategies, including hand 


signals, to gain students’ attention at 6 of the 
elementary schools but not seen at middle/high 
schools. 


* Few observations of students misbehaving. 


Teachers used a variety of techniques to handle 
behavior. 


“Checks” or “Fouls.” 


Capturing Kids’ Hearts 31 


Wake County Public School System 


Anisa Rhea, Ph.D. and Malkeet Singh, Ph.D. 


Data, Research, and Accountability Department 
Wake County Public School System 
Cary, North Carolina 


Acknowledgements 


The authors would like to acknowledge the 
contributions their DRA colleagues made to this 
report, specifically for their assistance conducting 
focus groups and classroom observations. 


Capturing Kids’ Hearts 32 


